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Preface 



Properly conceived and implemented new initiatives are essential to 
improving schools and communities. Such new initiatives usually are pursued 
as projects, pilots, or demonstrations, with temporary funding and staffing. 
When the funding ends, more often than not much of what has been 
developed disappears. Sometimes this is appropriate (e.g., when what was 
developed turns out not to be effective or important). At other times, the loss 
represents a set back for all stakeholders. 



Optimally, plans should be made at the onset with respect to sustaining 
valuable functions developed during the special funding period, and 
sustainability should be a focus from day one of implementation. With most 
such work, however, the pressure of just becoming operational often means 
that sustainability is not a major focus until well into the second year of a three 
year funding period. Moreover, most staff have not be prepared to pursue their 
work in ways that maximize sustainability and scale-up. 



This brief is designed to provide a quick overview of basic ideas, phases, 
stages, and steps related to the planning, implementation, maintenance, and 
scale-up of valuable new initiatives. There is a particular emphasis on 
sustainability. 



The Center has several more detailed guides and reports that expand on 
these matters. For example, see Sustaining School and Community Efforts 
to Enhance Outcomes for Children and Youth: A Guidebook and Tool Kit. 
Available online at: 

http://smhp.psych.ucla.edu/pdfdocs/sustaining.pdf 



Howard Adelman & Linda Taylor 
Co-Directors 



We are confronted with 
insurmountable opportunities. 

Pogo 



The history of schools is strewn with valuable practices that 
were not sustained, never mind replicated. Naturally, financial 
considerations played a role, but a widespread “project 
mentality” also is culpable. This mind set leads to thinking 
about the work as time-limited and contributes to 
marginalizing its status and exacerbates trends toward 
fragmented approaches. It also works against the type of 
systemic changes that sustain and expand innovations. 

Optimally , sustainability should be a focus 

from the day a project is implemented. 
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Ulitmately, only three things matter about educational reform. Does it 
have depth: does it improve important rather than superficial aspects 
of students’ learning and development? Does it have length: can it be 
sustained over long periods of time instead of fizzling out after the first 
flush of innovation? Does it have breadth: can the reform be extended 
beyond a few schools, networks or showcase initiatives to transform 
education across entire systems or nations? 



Andy Hargreaves & Dean Fink 





New Initiatives: Considerations Related to Planning, 
Implementing, Sustaining, and Going-to-Scale 



If we want to bring . . . quality , equity , and new life to our system - 
we must trust in a vision and a process of change. 

Dwight Allen 



The difficult work of implementing and sustaining any major innovation involves a host 
of complementary activity. This is particularly so with respect to developing and 
maintaining school-community collaboration. 

Increasingly, it is becoming evident that schools and communities (including institutions 
of higher education) should work closely together in order to generate the systemic 
changes necessary for meeting their mutual goals. While informal linkages are relatively 
simple to acquire, establishing major systemic reforms involves complicated long-term 
connections, especially when the goal is to strengthen youth, their families, and the 
community. Achieving such goals requires vision, cohesive policy, leadership, and an 
appreciation of the processes involved in planning, implementing, sustaining, and going- 
to-scale. 

This document offers a brief discussion about these matters to increase awareness of 
what is involved. 

Introduction 

With a view to sustaining valued functions, most demonstration projects and initiatives can be 
a catalyst for systemic change. More to the point, it is frequently the case that such projects 
must produce systemic changes or much of what they have developed is unlikely to be 
sustained. Federally-funded projects, such as those established through the Safe 
Schools/Healthy Students initiative, illustrate both the need and opportunity for being a 
catalytic force. These projects are funded with the aim of coalescing school and community 
collaboration for violence prevention. As the first cohort of projects entered their third and final 
year of federal support, the scramble began to find another grant to sustain threatened functions. 
Much earlier, a few projects realized that sustainability should not be thought about in terms 
of hopefully finding more grant money. Rather, they understood the necessity of taking steps 
each year to move policy in ways that would sustain valued functions that had been established 
through the project’s work. Moreover, they understood the importance of embedding such 
functions in a broader context to enhance their status in the eyes of policy makers. 

Those projects that pursued the categorical agenda of improving violence prevention mainly 
took the tack of adding on some services and programs. Although local policy makers tend to 
be pleased that such projects bring in added resources, they also view the work in terms of the 
limited categorical emphasis and seldom integrate the project’s services and programs into 
school improvement planning. This contributes to the fragmentation and marginalization that 
characterizes school and community efforts to address the many barriers to learning and 
teaching and works against sustaining the innovations when the project ends. 
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Projects as Catalysts for Systemic Change 

To counter the tendency for project functions to be viewed as having limited value, staff must 
strive to reframe the work and find their way to key decision making tables. This encompasses 
placing the activity into a broader context in terms of intervention focus, for example, reframing 
the activity so that it is seen as an integral part of a comprehensive, multifaceted, and cohesive 
approach to enhancing the school’ s ability to meet its mission. It calls for negotiating to be fully 
included in prevailing decision making, capacity building, and operational infrastructures in 
order to effect decisions and work toward reversing existing fragmentation and marginalization. 
It involves engaging decision makers in discussion of the feasibility of replicating the work on 
a large scale, combined with that of others, to enhance intervention effectiveness for many, not 
just a few, students and families. By working in this way, project staff position themselves to 
be a catalytic force. 



Escaping Project Mentality 

For projects to play a catalytic role for systemic change, staff must be mobilized to do so. And, this 
requires overcoming the phenomenon that has been dubbed “project mentality.” 

Project Mentality is a Barrier. 

A common tendency is for those involved in a project to think about (a) their work as simply a specially 
funded project and (b) their jobs as providing project-based discrete services. It also is common for 
policy makers and those interacting with project staff to assume the work being done will end when the 
grant runs out. It is not surprising, then, that everyone sees the new activity mainly in narrow and time- 
limited terms. This mind set contributes to fragmented approaches and marginalized status and, thus, 
works against developing comprehensive, multifaceted, and integrated programs for enhancing long- 
term positive results for school and community. It also works against capitalizing on the opportunity to 
be a catalyst for the type of systemic changes that sustain and expand innovations. 

Moreover, as the funding cycle nears its end, a number of very human concerns make it difficult for staff 
to focus on systemic change as the key to sustaining valued functions. These concerns include fear - of 
program elimination and job loss and belief that extramural funding is the only hope. These concerns 
push project staff to pursue a limited strategy for sustainability - seeking additional, dedicated funding 
to continue as a categorical project, rather than focusing on systemic changes that can incorporate 
valuable innovations. 



Strategies for Overcoming Project Mentality 

Overcoming project thinking begins with redefining the work. Rather than allowing it to be seen as a 
1, 2, or 3 year project, it should be reconceived as an ongoing initiative. After receiving a grant, we 
recommend never again referring to the work as a “project.” Next, it is wise to establish a potent steering 
body (not a figure-head advisory board). Such a body should consist of influential champions for the 
initiative and other individuals who arc highly committed to steering the staff in ways that not only 
achieve immediate objectives, but can catalyze systemic changes. 

Early in the first year, the plan detailed in the project proposal should be moiphed into a strategic plan 
for the ongoing initiative. This evolved plan should cover at least two years of activity beyond the 
funding period and should delineate, for each year, plans related to sustainability. As early as feasible, 
the steering body should push for adoption by policy makers of the full strategic plan. 

An ongoing strategic concern involves enhancing staff motivation and capability to play a catalytic role. 
The complexity of building their capacity requires guidance and support from professionals with 
mastery level competence for creating a climate for change, facilitating change processes, and 
establishing an institutional culture of continuous learning. 
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I. Toward Understanding Sustainability as a 
Systemic Change Process 

There is growing interest in understanding how to sustain effective 
innovations and some research related to evaluating sustainability (e.g., 
Woodbridge & Huang, 2000; Century & Levy, 2002; Trickett, 2002). Our 
approach to sustainability has evolved over many years, first in 
connection with trying to sustain demonstration programs, then as part 
of efforts to replicate innovations on a large-scale (see Adelman & 
Taylor, 1997a; Taylor, Nelson, & Adelman, 1999). Confronted with the 
problems and processes of scale-up, we generated a broad working 
framework of major considerations relevant to planning, implementing, 
and sustaining innovative approaches and going-to-scale. (Note: The 
process of large-scale replication often is called diffusion, replication, roll 
out, going-to-scale, or scale-up; we use the terms interchangeably here.) 

A dictionary definition indicates that to sustain is 

to keep in existence; to maintain; 



Another way to view sustainability is in terms of institutionalizing system changes. As 
Robert Kramer states: 

Institutionalization is the active process of establishing your initiative - not 
merely continuing your program, but developing relationships, practices, 
and procedures that become a lasting part of the community. 

Few will argue with the notion that something of value should be sustained if it is 
feasible to do so. Thus, the keys to sustainability are clarifying value and demonstrating 
feasibility. Both these matters are touched upon on the following pages. 



to nurture; to keep from failing; to endure 




The Board meeting is called to order: 
the problem for today is whether 
to hire 3 security guards or 
2 teachers 
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A. Sustainability of What? Making a Strong Argument 



One of the most pressing concerns to the staff of a specially funded project is sustaining their 
jobs when the project ends. The desire for maintaining one’s job is more than understandable. 
The problem is that this is the weakest argument for sustainability that can be offered to 
decision makers, especially when budgets are tight. Policy makers are constantly confronted 
with requests to maintain and add more personnel. Their decisions are supposed to be based 
on evidence of need and institutional priorities. For this reason, requests that simply advocate 
for sustaining all facets of a complex and expensive project also are weak. Decision makers 
want to know which facets are really necessary to achieve outcomes and which are nice but 
unessential accessories. 



Two Alternative Ways of Thinking about Sustaining Programs 

(1) Give us more money so we can carry on the work. 

(2) We need to make systemic changes because if we don’t we will lose 

some valued functions. 

With respect to alternative 1 , the focus often is on writing for grants, providing 
services that tap into third party payers (e.g., Medicaid), fund raising 
campaigns, or convincing school and/or agency decision makers to allocate 
money to cover personnel. More often than not, these efforts do not provide 
the needed resources. Thus, as the end grows near, there is a growing 
realization and a sinking feeling that much of the activity and most of the staff 
cannot be maintained. 

With respect to alternative 2, it is recognized from the onset that sustainability 
of valued functions requires making and institutionalizing systemic changes. 
This involves (a) creating readiness for such changes and (b) playing an 
active role in guiding implementation of the changes. 

While these alternatives are not mutually exclusive, it is wise to begin with 
the second. That is, it is best to think in terms of the probability that more 
money will not be available when current funding runs out. This means 
moving away from a project mentality and then connecting the activity to 
critical system needs and using resources to leverage systemic changes. In 
schools these days, connecting with system needs means fully integrating 
the work into the accountability demands of the No Child Left Behind Act. 
Leveraging includes using allocated funds as a catalyst and also cultivating 
champions (including key district and school leaders). 






Strong arguments for sustaining school-based innovations are framed within a “big 
picture” context of school and community efforts to strengthen students, families, 
schools, and/or neighborhoods. Compelling arguments (a) focus on specific 
functionsXhaX are essential to achieving highly valued outcomes and that will be lost 
when a project ends, (b) connect those functions with the overall vision and mission 
of the institutions asked to sustain them, and (c) clarify cost-effective strategies for 
maintaining the functions. 



For example, in our work developing innovations to better meet the needs of 
students experiencing learning, behavior, and emotional problems, we always stress 
how often the educational mission is thwarted because of many factors that interfere 
with youngsters’ learning and performance. We also emphasize that, if schools are 
to ensure that all students succeed, designs for school improvement must reflect the 
full implications of educating all students. Clearly, all includes more than students 
who are motivationally ready and able to profit from “high standards” demands and 
expectations. Thus, the focus on all must also include the many who aren’t 
benefitting from instructional reforms because of a host of barriers interfering with 
their development and learning, including external risk factors arising from 
neighborhood, family, school, and peer determinants and internal conditions such 
as those related to biological and psychological dysfunctioning. We remind policy 
makers that ensuring all students have an equal opportunity to succeed at school 
is the reason schools invest in education support programs and services and that 
given how substantial the investment is, greater attention must be paid to rethinking 
learning supports. From this perspective, we offer the umbrella of a comprehensive, 
multifaceted enabling or learning support component to coalesce the full range of 
functions that can address such barriers. The emphasis on addressing barriers to 
student learning allows us to present and underscore why new approaches are 
needed; in particular, we stress the need to fill basic gaps in the ability of schools 
to engage and re-engage students in effective classroom learning. Finally, we 
discuss cost-effectiveness by focusing on reducing fragmentation and enhancing 
resource use via systemic changes related to restructuring how existing student 
supports are conceived and implemented (Adelman, 1996; Adelman & Taylor, 
1997b; Adelman, Taylor, & Schneider, 1999; Center for Mental Health in Schools, 
1999). The following sections amplify on the above points. 
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Presenting a strong argument that there is something of value to sustain begins with 
understanding what is likely to be a weak argument. 

Weak arguments. One of the most pressing concerns to the staff of a specially 
funded project is sustaining their jobs when the project ends. The desire for 
maintaining one’s job is more than understandable. The problem is that this is the 
weakest case that can be made for sustaining a program. Also weak is any argument 
that advocates for sustaining all facets of a complex and expensive program. 
Decision makers want to know which facets are really necessary to achieve 
outcomes and which are nice but unessential accessories. 

Strong arguments focus on specific functions that are essential to 
achieving highly valued outcomes and that will be lost when a project 
ends. 

Strong arguments connect the functions to be sustained with the overall 
vision and mission of the institutions that are being asked to sustain them 
and clarify cost-effective strategies for doing so. 

Strong arguments are framed within a “big picture” context of school and 
community efforts to (a) address barriers to development and learning 
and (b) promote healthy development.* 



*See the Center document: Sustaining School and Community Efforts to 
Enhance Outcomes for Children and Youth: A Guidebook and Tool Kit. 
Available online at:http://smhp.psych.ucla.edu/pdfdocs/sustaining.pdf 

Part II of that document offers some tools for use in clarifying the current 
status of the local “big picture” context. The tools reflect the growing 
understanding that schools and communities (including institutions of higher 
education) must work closely together in order to meet their overlapping goals. 
Parts II, IV, & V focus on strong arguments. 

In Parts IV and V, the discussion stresses that, while informal school and 
community linkages are relatively simple to acquire, establishing major long-term 
connections is complicated, especially when the goal is to strengthen youth, their 
families, and the community. Achieving such goals requires vision, cohesive 
policy, leadership, and a relentless commitment to sustaining key functions and 
structural mechanisms. 
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B. What’s involved in sustaining valued functions? 



Sustainability involves a host of complementary activities. The figure on the following page 
can be used as a framework for understanding major matters for consideration in planning, 
implementing, sustaining, and going-to-scale. It also can be used as a template for 
establishing benchmarks for purposes of formative evaluation (see Appendix C). As the 
figure illustrates, changes may encompass introducing one or more interventions, 
developing a demonstration at a specific site, or replicating a prototype on a large-scale. 



Whatever the nature and scope of focus, all the key facets outlined in the figure come into 
play. 



Each cell in the matrix warrants extensive discussion. Here, we must limit ourselves to 
highlighting the host of interacting concerns and activities involved in sustaining valued 
initiatives. 

(1) With respect to sustainability, the nature and scope of focus raises such questions 

as: What specific functions will be implemented and sustained? Will one or more 
sites/organizations be involved? Is the intent to make system-wide changes? 

(2) With respect to key facets, whatever the nature and scope of the work, efforts for 

sustainability begin with articulation of a clear, shared vision for the initiative, 
ensuring there is a major policy commitment from all participating partners, 
negotiating partnership agreements, and designating leadership. This is followed 
by processes for enhancing/developing an infrastructure based on a clear 
articulation of essential functions, including mechanisms for governance and priority 
setting, steering, operations, resource mapping and coordination. Pursuing the 
work requires strong facilitation related to all mechanisms, redeploying resources 
and establishing new ones, building capacity (especially personnel development 
and strategies for addressing personnel and other stakeholder mobility), and 
establishing standards, evaluation processes, and accountability procedures. And, 
throughout, there must be an ongoing focus on social marketing. 

(3) When sustainability is approached as systemic change, the process must address 

each of the major phases of systemic change. These include (a) creating readiness 
with respect to the climate/culture for change by enhancing both the motivation and 
capability of a critical mass of stakeholder, (b) initially implementing changes by 
phasing them in with well-designed guidance and support, (c) maintaining and 
sustaining changes through practices that ensure institutionalization, and (d) 
ensuring appropriate evolution by enabling stakeholders to become a community 
of learners and facilitating periodic creative renewal activity. 
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Figure 1. New Initiatives: 

Considerations Related to Planning, Implementing, Sustaining, and Going-to-Scale 
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Exhibit 1 

Summary of Some Specific Concerns Related to 
Sustainability Planning and Implementation 

(1) Nature and scope of focus 

- What specific functions are to be sustained (e.g., specific interventions or 
program packages) 

- Will one or more sites/organizations be involved? 

- Is the intent to make system-wide changes? 

(2) Key facets related to undertaking any area of focus 

- Ongoing social marketing 

- Articulation of a clear, shared vision for the work 

- Ensuring there is a major policy commitment from all participating partners 

- Negotiating partnership agreements 

- Designating leadership 

- Enhancing/developing an infrastructure based on a clear articulation of essential 
functions (e.g., mechanisms for governance and priority setting, steering, 
operations, resource mapping and coordination; strong facilitation related to all 
mechanisms) 

- Redeploying resources and establishing new ones 

- Building capacity (especially personnel development and strategies for 
addressing personnel and other stakeholder mobility) 

- Establishing standards, evaluation processes, and accountability procedures 

(3) Phases related to making systemic changes 

- Creating readiness (motivation and capability - enhancing the climate/culture 
for change) 

- Initial implementation (phasing-in the new with well-designed guidance and 
support) 

- Institutionalization (maintaining and sustaining the new) 

- Ongoing evolution and creative renewal 
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In discussing approaches for sustaining “community schools,” the Coalition for 
Community Schools (2000) offers a range of “principles.” In particular, the 
Coalition highlights the importance of policies and practices that 

- use school-community teams at the site level to 
integrate resources and strategies 

- honor and encourage existing school-community governance 
arrangements 

- support local decision making 

- improve coordination of funding streams 

- build infrastructure 

- negotiate joint-use agreements 

- strengthen pre-service and in-service development 

- support inter-professional initiatives 

- create and sustain capacity-building organizations 
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C. Guidelines, Stages, and Steps 



As indicated in Figure 1 , the phases of the change process are a major dimension 
of the framework. Although these phases are rather self-evident, the intervention 
steps related to sustaining valued functions are less so. As a guide for those 
working on sustainability and system change, we have drawn on a what we have 
learned from the literature and our own work to delineate 1 6 key steps related to the 
first two phases of the change process (i.e., creating readiness and initial 
implementation). These are organized into four “stages.” The stages are conceived 
in terms of the need to intervene in ways that 1) develop a strong argument for 
sustaining functions, 2) mobilize interest, consensus, and support among key 
stakeholders, 3) clarify feasibility, and 4) proceed with specific systemic changes to 
sustain innovations. These stages and steps are offered below as guides for specific 
action planning. 



Below, we highlight 16 steps (organized into four “stages”). Part II offers concrete 
examples and some specific tools and aids related to each step. 



Remember: The following formulation of stages and steps is designed to guide 
thinking about sustainability and systemic change. It is not meant as a rigid format 
for the work. An overriding concern in pursuing each step is to do so in ways that 
enhance stakeholders’ readiness, especially motivational readiness. A particularly 
persistent problem in this respect is the fact that stakeholders come and go. There 
are administrative and staff changes; some families and students leave; newcomers 
arrive; outreach brings in new participants. The constant challenge is to maintain the 
vision and commitment and to develop strategies to bring new stakeholders on 
board and up-to-speed. Addressing this problem requires recycling through capacity 
building activity in ways that promote the motivation and capability of new 
participants. 

When a broad range of stakeholders are motivated to work together to sustain 
progress, they come up with more creative and effective strategies than any manual 
can prescribe. Thus, while concepts and procedures are invaluable guides, building 
a cadre of stakeholders who are motivationally ready and able to proceed is the first 
and foremost consideration. The necessary motivation comes from the desire to 
achieve better outcomes; it comes from hope and optimism about a vision for what 
is possible; it comes from the realization that working together is essential in 
accomplishing the vision; it comes from the realization that system changes are 
essential to working together effectively. And, maintaining motivation for working 
together comes from valuing each partner’s assets and contributions. 



11 







First, a few guidelines for pursuing sustainability as systemic change: 

- To counter marginalization, translate interventions into functions that are essential to the 
institution’s mission and accountability measures and frame them in terms of a comprehensive 
approach. 

- To avoid fragmentation and counterproductive competition among staff, design and implement 
new and expanded school-based activities in ways that integrate them fully with existing school 
programs, services, and personnel. 

- Use acquisition of extra-mural funding to leverage commitments for the type of systemic 
changes that will be essential to sustaining and scaling-up valued functions. (In doing so, 
establish clear priorities, and revisit memoranda of understanding - MOUs - to leverage 
stronger commitments.) 

- Focus first on the redeployment of current resources so that recommendations for systemic 
change are based on existing resources as much as is feasible. (This requires mapping and 
analyzing the available resource base.) Requests for additional resources are made only after 
it is evident that major gaps cannot be filled using existing resources more efficiently. 

- Design and establish an infrastructure that not only can carry out program functions, but also 
connects with decision making bodies and is capable of facilitating systemic change. For 
example, someone must be responsible for facilitating the creation of motivational readiness 
for any specific systemic change. 

- Use effectiveness data and information on cost-effectiveness in advocating for sustaining 
specific activities and approaches. 

- Identify a critical mass of “champions” to advocate and expedite and establish them as an 
active steering body. 

- Throughout, pursue social marketing and formative and benchmark evaluation. 



Stage A: Preparing the Argument for Sustaining Valued Functions 

The process of preparing a strong argument for sustainability begins by ensuring that 
advocates for sustaining a project’s functions understand the larger context in which such 
functions play a role (see Part II). Of particular importance is awareness of prevailing and 
pending policies, institutional priorities, and their current status and how existing resources 
might be redeployed to sustain valued functions that otherwise will be lost. With this in mind, 
there are five steps to pursue in readying the argument: 

1. Developing an understanding of the local “big picture” context for all relevant 

interventions. This involves, for example, amassing information that clarifies the 
school and community vision, mission statements, current policies, and major 
agenda priorities. 

2. Developing an understanding of the current status of efforts to accomplish goals 

related to the school and community vision, for example, clarifying the degree to 
which current priorities are well-founded and the rate of progress toward addressing 
major problems and promoting healthy development. 
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3. Delineating the functions, tasks, and accomplishments the project initiative has 

contributed with respect to the larger agenda and where the functions fit in terms 
of current policy and program priorities. 

4. Clarifying what functions will be lost if the school(s) and community do not determine 

ways to sustain them. The emphasis here is on articulating the implications of the 
loss in terms of negative impact on achieving the larger agenda. 

5. Articulating cost-effective strategies for sustaining functions, for example, focusing 

on how functions can be integrated with existing activity and supported with existing 
resources, how some existing resources can be redeployed to sustain the 
functions, how current efforts can be used to leverage new funds. 



Stage B: Mobilizing Interest, Consensus, and 
Support among Key Stakeholders 

In presenting the argument for sustainability, it is important to have a critical mass of influential and 
well-informed stakeholders who will be potent advocates for the initiative. The steps involved in 
developing this cadre of supporters include: 

6. Identifying champions and other individuals who are committed to sustaining the 

functions and clarifying the mechanism(s) for bringing supporters together to steer 
and work for sustainability. 

7. Planning and implementing a “social marketing” strategy to mobilize a critical mass 

of stakeholder support. 

8. Planning and implementing strategies to obtain the support of key policy makers, 

such as administrators and school boards. 

Stage C: Clarifying Feasibility 

The preceding steps all contribute to creating initial readiness for making decisions to sustain 
valued functions. Next steps encompass formulating plans that clarify specific ways the functions 
can become part of the larger school and community agenda. This raises considerations related 
to infrastructure and daily operations and the full range of systemic change concerns. These are 
addressed by: 

9. Clarifying how the functions can be institutionalized through existing, modified, or 

new infrastructure and operational mechanisms, for example, mechanisms for 
leadership, administration, capacity building, resource deployment, and 
integration of efforts. 

10. Clarifying how necessary changes can be accomplished, for example, 
mechanisms for steering change, external and internal change agents, and 
underwriting for the change process. 

1 1 . Formulating a longer-range strategic plan for maintaining momentum, progress, 
quality improvement, and creative renewal. 
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By this point in the process, the following matters should have been clarified: 

(a) what valued functions could be lost, (b) why they should be saved, and (c) who can help 
champion a campaign for saving them. In addition, strong motivational readiness for the 
necessary systemic changes should have been established. Done effectively, the process 
will have engendered strong motivational readiness for the necessary systemic changes. 



Stage D: Proceeding with Specific Systemic Changes 

At this juncture, it is time to initiate the implementation process for the necessary systemic changes. 
Because substantive change requires stakeholder readiness, it is essential to determine if the 
preceding steps accomplished the task. If not, it becomes necessary to revisit some of the earlier 
steps. Then, it is a matter of carrying out the plans made during Stage C with full appreciation of 
the complex dynamics that arise whenever complex systems undergo change. Specific steps 
encompass: 

12. Assessing, and if necessary enhancing, readiness to proceed with systemic 
changes needed to sustain valued functions.. 

13. Establishing an infrastructure and action plan for carrying out the changes. 

14. Anticipating barriers and how to handle them. 

15. Negotiating initial agreements, such as a memorandum of understanding. 

16. Maintaining high levels of commitment to accomplishing necessary systemic 
changes, for example, ensuring each task/objective is attainable, ensuring 
effective task facilitation and follow-through, negotiating long-term agreements 
and policy, celebrating each success, and facilitating renewal. 

Clearly, the many steps and tasks described above call for a high degree of commitment 
and relentlessness of effort. Major systemic changes are not easily accomplished. 
Awareness of the myriad political and bureaucratic difficulties involved in making major 
institutional changes, especially with limited financial resources, leads to the caution that 
the type of approach described above is not a straight-forward sequential process. Rather, 
the work proceeds and changes emerge in overlapping and spiraling ways. 

Do you understand the questions? 

/ 



It’s the answers that are hard! 

/ 






In discussing how the integrity of initiatives gets lost, Denise 
Gottfredson (2001) states: 

“...a greater degree of implementation integrity can be 
expected with explicit, user-friendly innovations for 
which a great deal of training is offered and when 
(staff) perceive that the innovation meets a need and 
have participated in the planning for the innovation. If 
the program is complex, more effort must be expended 
to increase clarity and perceived need. Greater integrity 
can be expected in schools that have highly skilled 
(staff) who communicate well and have high sense of 
self-efficacy, cultural norms that do not reject the 
innovation, strong district- and school-level leadership, 
staff stability, central office support, and a climate 
supporting change (e.g., problem-solving focus, high 
staff morale and commitment to change, no history of 
failed implementation, and a relatively low level of 
turbulence). Finally, local adaptation is likely to occur, 
especially with more complex programs. This 
adaptation process, although necessary has the potential 
to alter the program drastically.” 

Gottfredson goes on to stress that school innovations are 
implemented in “the contexts of local school districts, state education 
agencies, and federal government policies, practices, and funding 
streams, and are influenced by local politics and community 
pressures.” And, because of various inequities, she sees urban, inner- 
city settings as special cases with additional concerns to be 
addressed. 
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II. Going to Scale 



Efforts to create systemic changes require much more than implementing demonstrations at 
a few sites. Improved approaches are only as good as the ability of a school district and 
community to develop and institutionalize them on a large scale. This process often is called 
diffusion, replication, roll out, or scale-up. 



For the most part, education and community researchers and reformers have paid little 
attention to the complexities of large-scale diffusion. This is evident from the fact that the 
nation’s research agenda does not include major initiatives to delineate and test models for 
widespread replication of education reforms (see Replication and Program Services, 1993; 
Taylor, Nelson, & Adelman, 1999; Vander Ark, 2002). Furthermore, leadership training has 
given short shrift to the topic of scale-up. Thus, it is not surprising that the pendulum swings 
that characterize shifts in the debate over how best to teach reading are not accompanied 
with the resources necessary to accomplish prescribed changes throughout a school-district 
in an effective manner. Common deficiencies include inadequate strategies for creating 
motivational readiness among a critical mass of stakeholders, especially principals and 
teachers, assignment of change agents with relatively little specific training in facilitating 
large-scale systemic change, and scheduling unrealistically short time frames for building 
capacity to accomplish desired institutional changes. As Tom Vander Ark, executive director 
of education for the Bill and Melinda Gates Foundation notes: “Effective practices typically 
evolve over a long period of high-functioning, fully engaged systems.” 



For many years, our work revolved mainly around developing demonstration programs. 
Major examples include the Early Assistance for Students and Families project (funded by 
the U.S. Department of Education), the restructuring of education support services in a large 
school district, and the development of the Urban Learning Centers' model for 
comprehensive school reform (supported by the New American Schools Development 
Corporation, NASDC). More recently, we have moved into the world of replicating new 
approaches on a large-scale. Confronted with the problems and processes of scale-up, we 
analyzed a broad range of psychological and organizational literature and delineated the 
following working framework for scale-up (Adelman & Taylor, 1997). 



Figure 2 lists specific scale-up oriented tasks related to the four phases of systemic change. 
(For more on each phase and a discussion of some major lessons learned from our recent 
efforts, see Adelman and Taylor, 1997 and Taylor, Nelson, & Adelman, 1999.) Each task 
requires careful planning based on sound intervention fundamentals. At the onset, we should 
stress that initiating and guiding prototype replication requires a scale-up mechanism. One 
way to conceive such a mechanism is in terms of a scale-up project. Such a project provides 
a necessary organizational base and skilled personnel for disseminating a prototype, 
negotiating decisions about replication, and dispensing the expertise to facilitate scale-up. 
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Figure 2. Prototype implementation and scale-up: Phases and major tasks. 



Phase I 

Creating Readiness: 

Enhancing the 
Climate/Culture 
for Change 



$ 



Phase II 

Initial 

Implementation: 

Adapting and Phasing- 
in the Prototype with 
Well-Designed 
Guidance and Support 

$ 



Phase III 

Institutionalization: 

Ensuring the 
Infrastructure 
Maintains and 
Enhances 

Productive Changes 



$ 



Phase IV 

Ongoing Evolution 



System Change Staff 

1 . Disseminates the prototype 
to create interest (promotion 
and marketing) 

2. Evaluates indications of 
interest 

3. Makes in-depth 
presentations to build 
stakeholder consensus 

4. Negotiates a policy 
framework and conditions of 
engagement with sanctioned 
bodies 

5. Elicits ratification and 
sponsorship by stakeholders 



System Change Staff 
continues contact with 
Organization Leadership 

20. Facilitates expansion of 
the formative evaluation 
system (in keeping with 
summative evaluation needs) 

21. Clarifies ways to improve 
the prototype 

22. Compiles information on 
outcome efficacy 



Implementation Team 
works at site with 
Organization 
Leadership to 



6. Redesign the organizational 
and programmatic 
infrastructure 

7. Clarify need to add 
temporary mechanisms for the 
scale-up process 

8. Restructure time (the school 
day, time allocation over the 
year) 

9. Conduct stakeholder 
foundation-building activity 



10. Establish temporary 
mechanisms to facilitate the 
scale-up process 

1 1 . Design appropriate 
prototype adaptations 

12. Develop site-specific plan 
to phase-in prototype 



16. Institutionalize ownership, 
guidance, and support 

17. Plan and ensure 
commitment to ongoing 
leadership 

18. Plan and ensure 
commitment to maintain 
mechanisms for planning, 
implementation, and 
coordination 

19. Plan for continuing 
education and technical 
assistance to maintain and 
enhance productive changes 
and generate renewal 
(including programs for new 
arrivals) 



Team works at 
site with appropriate 
Stakeholders 

13. Plans and implements 
ongoing stakeholder 
development/empowerment 
programs 

14. Facilitates day-by-day 
prototype implementation 

15. Establishes formative 
evaluation procedures 



Organization Leadership 
works with Stakeholders 
in evolving the prototype 



Adapted from: H.S. Adelman & L. Taylor (1997). Toward a scale-up model for replicating new approaches to 
schooling. Journal of Educational and Psychological Consultation, 8, 197-230. 
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Think about the best model around for how schools can improve the way they address 
barriers to student learning. Assuming the model has demonstrated cost-effectiveness and 
that a school-district wants to adopt/adapt it, the first problem becomes that of how to 
replicate it, and the next problem becomes that of how to do so at every school. Or, in 
common parlance, the question is: How do we get from here to there?. 

Whether the focus is on establishing a prototype at one site or replicating it at many, the 
systemic changes can be conceived in terms of four overlapping phases: (1) creating 
readiness - by enhancing a climate/culture for change, (2) initial implemen tation - whereby 
change is carried out in stages using a well-designed guidance and support infrastructure, (3) 
institutionalization - accomplished by ensuring there is an infrastructure to maintain and 
enhance productive changes, and (4) ongoing evolution - through use of mechanisms to 
improve quality and provide continuing support. 

As indicated in the following figure, a change mechanism is needed. One way to conceive 
such a mechanism is in terms of a system implementation staff. Such staff provides a 
necessary organizational base and skilled personnel for disseminating a prototype, 
negotiating decisions about replication, and dispensing the expertise to facilitate 
implementation of a prototype and eventual scale-up. They can dispense expertise by 
sending out a team consisting of personnel who, for designated periods of time, travel to the 
location in which the prototype is to be implemented/replicated. A core team of perhaps two- 
to-four staff works closely with a site throughout the process. The team is augmented 
whenever a specialist is needed to assist in replicating a specific element of the prototype 
design. Implementation and scaling-up of a comprehensive prototype almost always requires 
phased-in change and the addition of temporary infrastructure mechanisms to facilitate 
changes. 

Figures 1 and 2 briefly highlight key facets and specific tasks related to the four 
phases of prototype implementation and eventual scale-up. Note in particular the 
importance of 

- ongoing social marketing 

- articulation of a clear, shared vision for the work 

- ensuring there is a major policy commitment from all participating partners 

- negotiating partnership agreements 

- designating leadership 

- enhancing/developing an infrastructure based on a clear articulation of essential 
functions (e.g., mechanisms for governance and priority setting, steering, 
operations, resource mapping and coordination; strong facilitation related to all 
mechanisms) 

- redeploying resources and establishing new ones 

- building capacity (especially personnel development and strategies for addressing 
personnel and other stakeholder mobility) 

- establishing standards, evaluation processes, and accountability procedures. 
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Systematic efforts to scale-up can dispense expertise by sending out a scale-up team 
consisting of project staff who, for designated periods of time, travel to replication sites. A 
core team of perhaps two-to-four staff works closely with a site throughout the replication 
process. The team is augmented whenever a specialist is needed to assist with a specific 
element, such as new curricula, use of advanced technology, or restructuring of education 
support programs. Scaling-up a comprehensive prototype almost always requires phased-in 
change and the addition of temporary infrastructure mechanisms to facilitate changes. 

Below are a few points about underwriting the change process. 



A basic funding principle is that no single source of or approach to financing 
is sufficient to underwrite major systemic changes. 

Opportunities to Enhance Funding For More Information 



-reforms that enable redeployment of existing funds 
away from redundant and/or ineffective programs 
-reforms that allow flexible use of categorical funds 
(e.g., waivers, pooling of funds) 

-health and human service reforms (e.g., related to 
Medicaid, TANF, S-CHIP) that open the door to 
leveraging new sources of MH funding 
-accessing tobacco settlement revenue initiatives 
-collaborating to combine resources in ways that enhance 
efficiency without a loss (and possibly with an increase) 
in effectiveness (e.g., interagency collaboration, public- 
private partnerships, blended funding) 

-policies that allow for capturing and reinvesting funds 
saved through programs that appropriately reduce costs 
(e.g., as the result of fewer referrals for costly services) 
-targeting gaps and leveraging collaboration (perhaps 
using a broker) to increase extramural support while 
avoiding pernicious funding 

-developing mechanisms to enhance resources through 
use of trainees, work-study programs, and volunteers 
(including professionals offering pro bono assistance). 



The Internet provides ready access to info on funding and 
financing. 

Regarding funding, see: 

>School Health Program Finance Project Database - 
http ://www2 . cdc .gov/nccdphp/shpfp/index. asp 
>School Health Finance Project of the National Conference 
of State Legislators - 

http://ncsl.org/programs/health/pp/schlfund.htm 
>Snap shot from SAMHSA - http://www.samhsa.gov 
>The Catalog of Federal Domestic Assistance - http://www.gsa.gov/ 
>The Federal Register - http://www.access.gpo.gov/GPOAccess 
>Gra«faWe/?-http://www. research, sunysb.edu/research/kirby.html 
>The Foundation Center - http://fdncenter.org 
>Surfin 'for Funds - guide to internet financing info 
http://smhp.psych.ucla.edu/ (search Quick Find) 

Regarding financing issues and strategies, see: 

>The Finance Project - http://www.financeproject.org 
>Center for Study of Social Policy - http://www.cssp.org 
>Center on Budget and Policy Priorities - http:www.cbpp.org 
>Fiscal Policy Studies Institute - http://www.resultsaccountability.com 



To foster service coordination, there are several ways to use existing dollars provided to a district by the 
federal government. For example, some districts use funds from Title I of the No Child Left Behind Act 
based on a provision that encourages steps to foster service coordination for students and families. A similar 
provision exists in the 1997 reauthorization of the Individuals with Disabilities Education Act. Other 
possible sources are Community MH Services block grant, funds related to after school programs, state- 
funded initiatives for school-linked services, etc. 
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########################## 

Awareness of the myriad political and bureaucratic 
difficulties involved in making major institutional 
changes, especially with limited financial resources, 
leads to the caution that the type of approach described 
above is not a straight-forward sequential process. 

Rather, the process proceeds and changes emerge in 
overlapping and spiraling ways. Nevertheless, it helps 
to have the type of step-by-step overview that has been 
just outlined. 

################################ 
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Some Helpful Websites to search related to Sustainability: 

The Community Toolbox - http://ctb.ku.edu/about/ 

The Financing Project - http://www.financeproject.org/ (several documents on topic) 
Northwest Regional Educational Laboratory - http://www.nwrel.org 
Annie E Casey Foundation - http://www.aecf.org (e.g., see documents such as “Rebuilding 
Communities” for discussions of sustainability) 

Calif. Center for Health Improvement - http://www.healthpolicycoach.org 



*A11 resources cited from the Center for Mental Health in Schools at UCLA are available online and can 
be downloaded from http://smhp.psych.ucla.edu 

In addition to those cited above, the following may be of interest: 

>Addressing Barriers to Student Learning & Promoting Healthy Development: 

A Usable Research-Base 

>Enhancing Classroom Approaches for Addressing Barriers to Learning: 

Classroom-Focused Enabling 

>Evaluation and Accountability: Getting Credit for All You Do 

>Expanding Educational Reform to Address Barriers to Learning: Restructuring Student 
Support Sendees and Enhancing School-Community Partnerships 
>Financial Strategies to Aid in Addressing Barriers to Learning 
> Financing Mental Health for Children & Adolescents (Brief and Fact Sheet) 

>Framing New Directions for School Counselors, Psychologists, & Social Workers 
>Integrating Mental Health in Schools: Schools, School-Based Centers, and 
Community Programs Working Together 

>New Directions in Enhancing Educational Results: Policymakers' Guide to Restructuring 
Student Support Resources to Address Barriers to Learning 
>New Initiatives: Considerations Related to Planning, Implementing, Sustaining, 
and Going-to-Scale 

>Organization Facilitators: A Change Agent for Systemic School and ommunity Changes 
>Resource-Oriented Teams: Key Infrastructure Mechanisms for Enhancing 
Education Supports 

> Re structuring Boards of Education to Enhance Schools' Effectiveness in Addressing Barriers 
to Student Learning 

>Sampling of Outcome Findings from Interventions Relevant to Addressing Barriers to Learning 
> School-Community Partnerships: A Guide 

>Working Together: From School-Based Collaborative Teams to School-Community-Higher 
Education Connections 
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Appendices 



A. Readiness for Systemic Change 

B. Social Marketing, Data, and Systemic Change 

C. Formative and Summative Evaluation of 

Efforts to Sustain Functions 
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Appendix A 

Readiness for Systemic Change 



Substantive change is most likely when high levels of positive energy among 
stakeholders can be mobilized and appropriately directed over extended 
periods of time. That is, one of the first concerns related to systemic change 
is how to mobilize and direct the energy of a critical mass of participants to 
ensure readiness and commitment. This calls for proceeding in ways that 
establish and maintain an effective match with the motivation and capabilities 
of involved parties. 



The initial focus is on communicating essential information to key 
stakeholders using strategies that help them understand that the benefits of 
change will outweigh the costs and are more worthwhile than competing 
directions for change. 



The strategies used must be personalized and accessible to the subgroups of 
stakeholders (e.g., must be “enticing,” emphasize that costs are reasonable, 
and engage them in processes that build consensus and commitment). 
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Benchmarks related to Creating Readiness for Systemic Change 
Related to School/Community Approaches to Addressing Barriers to Learning, 
Promoting Healthy Development, & Closing the Achievement Gap 



Location: 

I. Orienting Stakeholders 



Date Date 

Started Completed 



A. Basic ideas and relevant research base 
are introduced to key stakeholders 
using “social marketing'’ strategies 
>school administrators 
>school staff 

>families in the community 
>business stakeholders 



B. Opportunities for interchange are provided & 

additional in-depth presentations are made 
to build a critical mass of consensus for 
systemic changes 

C. Ongoing evaluation of interest is conducted until 

a critical mass of stakeholders indicate 
readiness to pursue a policy commitment 

D. Ratification and sponsorship are elicited from a 

critical mass of stakeholders 

II. Establishing Policy Commitment & Framework 

A. Establishment of a high level policy 

and assurance of leadership commitment 

A. Policy is translated into an inspiring vision, a 
framework, and a strategic plan that phases in 
changes using a realistic time line 

B. Policy is translated into appropriate resource 
allocations (leadership, staff, space, budget, time) 

D. Establishment of incentives for change 

(e.g., intrinsically valued outcomes, expectations 
for success, recognitions, rewards) 

E. Establishment of procedural options that reflect 
stakeholder strengths and from which those 
expected to implement change can select strategies 
they see as workable 

F. Establishment of an infrastructure and processes 
that facilitate change efforts 

G. Establishment of a change agent position 

H. Establishment of temporary infrastructure 
mechanisms for making systemic changes 

I. Initial capacity-building - developing essential 
skills among stakeholders to begin implementation 

J. Benchmarks are used to provide feedback on 
progress and to make necessary improvements 
in the process for creating readiness 
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Current 

Status 





Appendix B 

Social Marketing, Data, and Systemic Change 



Social marketing is an important tool for fostering a critical mass of 
stakeholder support for efforts to change programs and systems. 
Particularly important to effective marketing of change is the inclusion 
of the evidence base for moving in new directions. 

The handout included here can be used to provide a quick introduction 
as a basis for discussion by school-community partners about the 
importance of social marketing to sustainablity. 

For an example of a research base that can be used to support 
comprehensive, multifaceted approaches to addressing barriers to 
student learning, see the UCLA Center Brief entitled: Addressing 
barriers to student learning and promoting healthy development: A 
usable research base. This summary of data can be extrapolated and 
combined with local data and anecdotes to support a variety of school- 
community endeavors. The brief can be downloaded from the 
Center’s website: http://smhp.psych.ucla.edu - hard copies can be 
ordered at cost. 
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Appendix B 

Social Marketing as a Spiraling Facet of Program and Systemic Change 

Social marketing is a tool for accomplishing social change. 

As such, it can be used in good or bad ways. 

Social marketing draws on concepts developed for commercial marketing, but in the context of school 
and community change, we are not talking about selling products. We are trying to build a consensus 
for ideas and new approaches that can strengthen youngsters, families, and neighborhoods. Thus, we 
need to reframe the concept to fit our purposes. 

Some Basic Marketing Concepts as Applied to Changing Schools and Communities 

- the aim is to influence action by key stakeholders 

- to achieve this aim, essential information must be communicated to key stakeholders and 
strategies must be used to help them understand that the benefits of change will outweigh the 
costs and are more worthwhile than competing directions for change 

- the strategies used must be personalized and accessible to the subgroups of stakeholders (e.g., 
must be “enticing,” emphasize that costs are reasonable, and engage them in processes that 
build consensus and commitment) 

Because stakeholders and systems are continuously changing, social marketing is an ongoing process. 
Social Marketing as an Aid in Creating Readiness for Change 

From a teaching and learning perspective, the initial phases of social marketing are concerned with 
creating readiness for change. Substantive change is most likely when high levels of positive energy 
among stakeholders can be mobilized and appropriately directed over extended periods of time. That is, 
one of the first concerns related to systemic change is how to mobilize and direct the energy of a critical 
mass of participants to ensure readiness and commitment. This calls for proceeding in ways that establish 
and maintain an effective match with the motivation and capabilities of involved parties. 



With respect to systemic change, the initial aims are to 

- introduce basic ideas and the relevant research base to key stakeholders using “social marketing'’ strategies 

- provide opportunities for interchange & additional in-depth presentations to build a critical mass of 
consensus for systemic changes 

- conduct ongoing evaluation of interest until a critical mass of stakeholders indicate readiness to pursue a 
policy commitment 

- obtain ratification and sponsorship by critical mass of stakeholders 

- establish a high level policy and ensure leadership commitment 

- translate policy into an inspiring vision, a framework, and a strategic plan that phases in changes using a 
realistic time line 

- translate policy into appropriate resource allocations (leadership, staff, space, budget, time) 

- establish incentives for change (e.g., intrinsically valued outcomes, expectations for success, recognitions, 
rewards) 

- establish procedural options that reflect stakeholder strengths and from which those expected to implement 
change can select sttategies they see as workable 

- establish an infrastructure and processes that facilitate change efforts 

- establish a change agent position 

- establish temporary infrastructure mechanisms for making systemic changes 

- build initial implementation capacity - develop essential skills among stakeholders 

- use benchmarks to provide feedback on progress and to make necessary improvements in the process for 

creating readiness 
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Appendix C 



Formative & Summative Evaluation of Efforts to Sustain Functions 

As highlighted earlier, findings supporting the value of sustaining functions are 
invaluable in making the case for doing so. Such data come from intervention/program 
evaluation. 

In this appendix, the emphasis is on a different evaluation focus - monitoring and 
determining the efficacy of the sustainability activity. 

Essentially what is involved is: 

- formulating an evaluation action plan 

- adopting specific benchmarks for monitoring progress 

- specifying and measuring immediate indicators that functions are sustained 

- specifying and measuring longer-term indicators that functions are 
sustained 



On the following pages is a brief introduction to the evaluation problem. This 
is followed by a benchmark tool that can be adapted for local use in monitoring 
progress. The benchmarks are organized in terms of the four stages of the 
sustainability process as discussed in this brief and in other Center documents. 
(Note: In several instances, specific steps that logically go together are 
combined.) 



Immediate indicators that functions are sustained include data from several 
sources that show functions that were in danger of being lost (1) are being 
continued and (2) are being carried out in ways that maintain their potency. 



Longer-term indicators that functions are sustained include data from several 
sources that show functions in danger of being lost are institutionalized. The 
focus here is on matters such as including the functions in policy statements, as 
regular items in the budget, as part of regular job descriptions for 
administrative and line staff, as part of the systems’ accountability reviews, 
and so forth. 
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I. A Brief Overview of the Evaluation Problem 



Evaluation practiced at the highest level of the state-of-the-art is one means of 
speeding up the processes that contribute to human and social progress. 

Rossi, Freeman, & Wright 1 



Increased concern about evaluation in psychology and education has advanced the way evaluation 
is conceived. 2 Despite the breadth of this scholarly activity, widespread demands for 
accountability continue to narrow the way professionals, clients, policy makers, underwriters, and 
the general public think about evaluation. Social and political forces literally have shaped the 
whole enterprise of program evaluation. 3 

The prevailing cry is for specific evidence of efficacy — usually in terms of readily measured 
immediate benefits — and for cost containment. Although understandable in light of the unfulfilled 
promise of so many programs and the insatiable demands on limited public finances, such naive 
accountability demands ignore the complexities of intervention. The problem is well exemplified 
by the narrow focus found in reviews, analyses, and reanalyses of data on psychotherapy, behavior 
change, and early education programs. 4 

Besides responding to accountability pressures, two unfounded presumptions are at the core of 
most current evaluations in psychology and education. One premise is that an intervention in 
widespread use must be at a relatively evolved stage of development and thus warrants the cost 
of summative evaluation. The other supposition is that major conceptual and methodological 
problems associated with evaluating intervention efficacy are resolved. The truth, of course, is 
that interventions are frequently introduced prior to adequate development with a view to evolving 
them based on what is learned each day. Moreover, many well-institutionalized approaches 
remain relatively underfunded and underdeveloped. As to the process of evaluation, every review 
of the literature outlines comprehensive, unresolved concerns. Given this state of affairs, 
accountability demands are often unreasonable and chronically reflect a naive view of research 
and theory. 

Overemphasis on immediate evaluation of the efficacy of underdeveloped interventions draws 
resources and attention away from the type of intensive research programs necessary for advancing 
intervention knowledge and practice. Cost-effective outcomes cannot be achieved in the absence 
of cost-effective development of interventions and related intervention research. Premature 
efforts to carry out comprehensive summative evaluations clearly are not cost-effective. 
Consequently, policies mandating naive accountability run the risk of generating evaluative 
practices that are neither cost-effective nor wise. 5 

The evaluation problem, then, involves more than determining the efficacy of current interventions 
and more than finding better ways to evaluate efficacy. Broadly stated, it encompasses concerns 
about how to expand the focus of evaluation not only to contribute to improving practice, but also 
to aid in evolving theory and basic knowledge about intervention. 



On the following pages, our intent is to briefly highlight the concept of evaluation. 
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The Essence of Evaluation 



Evaluation involves determining the worth or value of something. 6 In formal terms, we define 
comprehensive evaluation as a systematic process designed to describe and judge an intervention's 
antecedents, transactions, and overall impact and value for purposes of making decisions and 
advancing knowledge. 7 

Everyone evaluates interventions with which they come in contact. Whenever anyone decides that 
an intervention is or isn't a good one, an evaluation is made. 8 Interveners judge whether their own 
and others' programs are going well. Clients are quick to formulate likes or dislikes of interveners 
and their programs. Administrators know which programs they think are working and which 
aren't. 

Some evaluative judgments simply reflect an individual's or group's informal observations. Other 
judgments are based on careful data gathering and analyses and use of appropriate sets of 
standards. Some evaluations only offer conclusions about the degree to which a program is 
effective. Most, however, also incorporate the conclusions of those judging the program in terms 
of whether they agree with what it is trying to do. Since what a program intends to do stems from 
its rationale, program evaluations inevitably influence views about the appropriateness of its 
underlying rationale. 

Systematic evaluation planning requires decisions about (1) the focus of evaluation (e.g., person 
or environment, immediate objectives vs. long-range aims), (2) whose perspective (e.g., client, 
intervener, program underwriter) is to determine the evaluation focus, methods, and standards 
used, and (3) the best way to proceed in gathering, analyzing, and interpreting information (e.g., 
specific measures, design). In making such decisions, concerns arise because what can be 
evaluated currently is far less than what a program may intend to accomplish. Furthermore, 
inappropriate bias and vested interests shape evaluation planning and implementation, thereby 
influencing whether a program is seen as good or bad. And all aspects of evaluation have the 
potential to produce negative effects; for instance, evaluation can lead to invasion of privacy and 
an undermining of the ability of clients and interveners to self-evaluate, and over time, what is 
evaluated can reduce and reshape a program's intended aims. 

Purposes. Intervention evaluation can aid efforts to (1) make decisions about whether to 
undertake, continue, modify, or stop an intervention for one or more clients and (2) advance 
knowledge about interventions in ways that can advance understanding of and improve practices 
(including utility), training, and theory. Evaluation is useful in relation to a great variety of 
interventions as an aid in assessing efficiency, effectiveness, and impact. As Rossi and Freeman 
state: 

The mass communication and advertising industries use fundamentally the same approaches in 
developing media programs and marketing products; commercial and industrial corporations 
evaluate the procedures they use in selecting and promoting employees and organizing their work 
forces; political candidates develop their campaigns by evaluating the voter appeal of different 
strategies; . . . administrators in both the public and private sectors arc continually assessing 
clerical, fiscal, and interpersonal practices of their organizations. The distinction between these 
uses of evaluation lies primarily in the intent of the effort to be evaluated ... to benefit the 
human condition . . . [or] for other purposes, such as increasing profits or amassing influence and 
power . 9 

Providing a broad categorical view of the areas in which evaluation is applied, Scriven outlines 
the "Big Six" plus others. The Big Six are listed as product, performance, personnel, program, 
proposal, and policy evaluations. To these, he adds two other applied fields. "The first is the 
evaluation of evaluations (meta-evaluation). . . . The second is a field comprising a set of fields: 
It might be called Tntradisciplinary evaluation,' the evaluation of the data, sources, explanations, 
definitions, classifications, theories, designs, predictions, contributors, journals, and so on within 
a discipline." Scriven concludes: "In toto, intradisciplinary evaluation is by far the largest part 
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of evaluation, and having practitioners do it with reasonable skills is the price of admission to the 
company of disciplines. Other applied fields besides the Big Six range from literary criticism and 
real estate appraisal to quality control in industry." 10 

Stake's evaluation matrix is reproduced in Figure 2 as an example of a framework designed to 
outline the general nature of information for meeting many evaluation purposes. 11 As the 
framework suggests, evaluation encompasses the acts of describing and judging an intervention's 
(1) rationale, including assumptions and intentions, (2) standards formaking judgments, (3) actual 
activity, including intended and unintended procedures and outcomes, and (4) costs — financial, 
negative effects, and so forth. To achieve the above ends in a comprehensive manner, both 
immediate and long-term information on an intervention must be gathered. 12 

Tasks For Planning. Awareness of tasks involved in planning an evaluation provides 
another perspective on the process. Such tasks reflect the necessity in evaluation planning of 
making decisions about the focus of the evaluation, its specific objectives, and appropriate 
methodology and measures. 

Our formulation identifies the following seven key planning tasks: 

Clarifying the intended use of information. Most important here is awareness of who 
wants the information and why they need it. Ultimately this translates into the 
question: What types of decisions are to be made? Also important is the matter of 
anticipating the use and political and motivational impact of evaluation processes and 
findings. This includes a significant appreciation of the often conflicting interests 
among the variety of interested parties (i.e., stakeholders). 

Understanding the intervention's rationale. In cases where evaluation includes 
judging the intervention rationale, pursuit of the above task (clarifying the intended use 
of evaluation information) will result in gathering information about the rationale. 
However, when the evaluation is designed with reference to a standardized set of 
objectives, clarification of the rationale becomes a separate task. In either case, an 
understanding of the intervention rationale can provide a separate basis for deciding 
about other intervention facets to evaluate. 

Formulating evaluation questions. Evaluative concerns are translated into a set of 
questions. For example: Were intended antecedent conditions present during the 

intervention? Which procedures were effective for which clients? Were there undesirable 
transactions? Were specific objectives achieved? Were long-range aims achieved? Did 
expected negative outcomes occur? Were there unexpected negative outcomes? 

A. Specifying information to be gathered. Relevant descriptive information that can answer 
each major question is specified. The more things one is interested in evaluating, the more 
one has to settle for samples of information. Some of the information likely will be of a 
quantitative nature; some may be qualitative. 13 

Specifying procedures. Decisions about information gathering are shaped first by what one 
wants to know and then are tempered by practical considerations. Problems related to 
gathering desired information become evident as one attempts to specify procedures. 
Limitations related to time, money, sample availability, valid measures, multivariate 
statistics, and personnel usually lead to major compromises in evaluation planning. For 
example, sometimes a good measuring instrument exists; sometimes only weak procedures 
are available; sometimes gathering desired information is not currently feasible. A special 
set of problems stems from the socio-political-economic concerns (e.g., threats to current 
status) and psychological reactance (e.g., fear-based resistance) that are common 
phenomena when evaluation is introduced. 14 
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Figure 2 

Layout of Statements and Data to Be Collected During Evaluation 



Descriptive matrix 

Intents Observations 



Judgment matrix 

Standards Judgments 



Underlying 

Intervention 

Rationale 



Antecedents 



Transactions 



Outcomes 



Source : R. Stake ( 1967). The countenance of educational evaluation. Teachers College Record , 68, 523-40. 
Reprinted with permission. 



Specifying a design. An evaluation design is used so that information can be gathered and 
interpreted appropriately. When someone asks how good an intervention is, judgments are 
based on the available information and are relative to some standard of comparison. A 
sound design ensures that appropriate bits of information (e.g., data) are gathered, 
including information for use as standards for judgments. A sound evaluation design also 
includes provision for the gathering and use of information for revising interventions as 
the process proceeds. 15 

Designating time and place for collecting information. Further practical considerations 
arise when evaluations are scheduled. The design sets the general parameters; the 
particulars are determined by practical factors such as resource availability. 

One major evaluation concern not reflected above involves decisions about the role of various 
interested parties. For example, as suggested throughout, rationales may differ with respect to 
what should be evaluated. If so, whose rationale should prevail? Every facet of an evaluation is 
influenced by the answer to this question. 
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Another matter not specifically addressed above involves ethical concerns associated with 
evaluation. Naturally, these are similar to those discussed in relation to assessment in general. 
For instance, evaluators must be concerned with how to minimize possible bias and conflicts of 
interest, as well as negative consequences that can arise from evaluation itself. 

Impact on Program Breadth. As the discussion to this point underscores, a common use of 
evaluation is to determine if one agrees with what the intervention is trying to accomplish and how 
well the intervention is accomplishing the full range of outcomes desired. The less a program is 
trying to achieve, the easier it is to determine these matters. It is hard to evaluate large-scale social 
programs, community agencies, and most school programs, for example, because they are trying 
to accomplish so many different goals. 16 

Ironically, the longer a program is subjected to external, formal evaluation, the less it may try to 
accomplish. At least this seems to be one negative effect of the big push toward behavioral and 
criterion-referenced outcomes as ways to improve accountability. That is, such approaches can 
cause a shift away from a program's long-range aims toward a limited set of immediately 
measurable objectives. This is a negative form of "teaching to the test" because, in the process, 
many important things are ignored simply because they will not be directly evaluated. 17 If one is 
not careful, the desire for information on effectiveness can redesign a program's underlying 
rationale in ways that inappropriately reduce its breadth of focus. 

Comprehensive evaluation should stress the full scope of desired intervention aims. That is, even 
when certain processes and outcomes are not easily measured, they still must be evaluated as well 
as is possible and kept in the forefront of discussions about a program's worth. For example: from 
a motivational perspective, a basic concern is whether a program enhances clients' interest, desire, 
and participation in improving their functioning. Because none of these outcomes is readily 
measured, the danger is that they will not be afforded the attention they warrant. 

In sum, evaluations of whether an intervention is any good must first address the question: Is what 
it is trying to accomplish appropriate? The frame of reference for such evaluations may be the 
intervention rationale or what others think the program should be doing or both. After judging the 
appropriateness of what is wanted or expected, a program's intended breadth of focus should guide 
efforts to evaluate effectiveness. Because not everything is measurable in a technically 
sophisticated way, some things will be poorly measured or simply reviewed informally. 
Obviously, this is less than satisfactory. Still, from a rational perspective, continued emphasis on 
the entire gamut of what is intended is better than limiting evaluation to approaches that 
inappropriately narrow the breadth of focus for intervention. 18 

In this context, we are reminded of Yankelovich's commentary on measurement: 

The first step is to measure whatever can be easily measured. This is okay as far as it goes. 
The second step is to disregard that which can't be measured or give it an arbitrary quantitative 
value. This is artificial and misleading. The third step is to presume that what can't be 
measured easily isn't very important. This is blindness. The fourth step is to say what can't be 
measured really doesn't exist. This is suicide. 19 
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NOTES 



1. P.H. Rossi, H.E. Freeman, & S. Wright (1979). Evaluation: A systematic approach (3rd ed.). 
Beverly Hills, CA: Sage. 

2. For a comparison of evaluation models, see D.F. Stufflebeam & W.J. Webster (1983). An analysis 
of alternative approaches to evaluation. In G.F. Madaus, M.S. Scriven, & D.F. Stufflebeam (Eds.), 
Evaluation models. Boston: Kluwer-Nijhoff; also see P.H. Rossi & H.E. Freeman (1989). Evaluation: A 
systematic approach (4th ed.). Newbury Park, CA: Sage. EJ. Posavac & R.G. Carey (1989). Program 
evaluation: Methods and case studies (3rd ed.). Englewood Cliffs, NJ: Prentice-Hall. For recent reviews 
of the topic, see W.R. Shadish, Jr., T.D. Cook, & F.C. Feviton (1991). Foundations of program evaluation: 
Theories of practice. Newbury Park, CA: Sage. F. Sechrest & A.J. Figueredo (1993). Program evaluation. 
Annual Review of Psychology, 44, 645-674. M. Scriven (1993). Hard-won lessons in program evaluation. 
San Francisco: Jossey-Bass. 

3. Recent reviews stress that the evolution of program evaluation in general and evaluation theory 
specifically has been shaped to a significant degree by evaluation researchers who were unprepared for their 
confrontations with complex social and political realities — including those associated with the shift from 
an industrial to a postindustrial (cybernetic) era. The demand for greater external validity has forced 
program evaluators to move beyond the prevailing paradigms and methods guiding the social sciences. For 
the most part, this demand reflects the socio-political-economic nature of intervention and evaluation. That 
is, interventions compete for limited societal resources and evaluation feeds into political decision making 
about which interventions are funded and levels of support. 

4. See F. Bond & B.E. Compas (Eds.) (1989). Primary prevention and promotion in the schools. 
Newbury Park: Sage, pp. 106-45). A. Kazdin (1990). Psychotherapy for children and adolescents. Annual 
Review of Psychology, 41, 21-54. M.J. Fambert, D.A. Shapiro, & A.E. Bergin (1986). The effectiveness 
of psychotherapy. In S.F. Garfield & A.E. Bergin (Eds.), Handbook of psychotherapy and behavior change 
(3rd ed.). New York: Wiley. A. Mitchell, M. Seligson, & F. Marx (1989). Early childhood programs and 
the public schools: Promise and practice. Dover, MA: Auburn House. R.E. Slavin, N.F. Karweit, & N.A. 
Madden (1989). Effective programs for students at risk. Boston: Allyn & Bacon. J.R. Weisz, B. Weiss, & 
G.R. Donnenberg (1992). The lab versus the clinic: Effects of child and adolescent psychotherapy. 
American Psychologist, 47, 1578-1585. 

5. Accountability pressures can lead to an overemphasis on immediate behavioral outcomes. Usually, 
decisions as to what and how to evaluate are made by those administering or funding an intervention. For 
example, with respect to specifying outcomes for evaluation, the primary focus in preparing IEPs for special 
education is on remedial outcomes. Furthermore, the prevailing emphasis is on specifying outcomes in 
terms of behavioral and criterion-referenced objectives. Similar trends are seen in psychology for 
interventions underwritten by third party payers. These trends no doubt are a major aid in efforts to 
evaluate whether outcomes are accomplished. However, the limited focus ignores the broader 
responsibility many interveners have for facilitating ongoing development and providing enrichment 
opportunities. A narrow focus on correcting problems also can be counterproductive to overcoming 
problems if the intervention involves little more than a set of laborious and deadening experiences. 
Moreover, many important facets of a program are not easily measured and thus may be given short shrift 
(e.g., self-concept, attitudes toward system improvement and problem solving). In general, the danger is 
that valuable intervention aims and goals are lost when all ends are specified in terms of highly concrete 
and easily measurable objectives. Not all complex long-range aims that an intervention should pursue can 
be stated as short-term or behavioral objectives. Indeed, only a relatively limited set of skills can be 
specified in highly concrete, behavioral terms — and even in these instances, it may not be desirable to do 
so for intervention puiposes. In education, beside the fact that specifying everything in this way would 
result in far too many objectives to teach, the Fend stresses teaching at the expense of learning. Moreover, 
attitudes, motivation, and creative functioning in the aits and sciences, for example, do not lend themselves 
to formulation in simple behavioral terms. 

The dilemmas raised by accountability pressures are well illustrated in an article on mental health 
services for children: see J.D. Burchard & M. Schaefer (1992). Improving accountability in a service 
delivery system in children’s mental health. Clinical Psychology Review, 12, 867-882. 
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6. We recognize the deficiencies of this simple definition. Still, it conveys the essence of the process. 
Reviewing the matter, Scriven states: "Evaluation is a process of determining certain evaluable properties 
of things, but there is more than one kind of such properties. Perhaps the most fundamental and important 
distinction among them is between merit or quality and worth or value." Using the example of a high 
school French teacher, he notes that the teacher may be the best in a school, but if enrollment patterns shift 
away from French, that teacher's worth or value to the school diminishes. The teacher’s merit (i.e., quality 
in terms of professional standards) has not declined, but his or her benefit (vis a vis meeting the school's 
needs) has. M. Scriven (1993). Hard-won lessons in program evaluation. San Francisco: Jossey-Bass, p. 
67. 

7. Rossi and Freeman use the terms evaluation and evaluation research interchangeably. Their 
definition states: "Evaluation research is the systematic application of social research procedures for 
assessing the conceptualization, design, implementation, and utility of social intervention programs." See 
P.H. Rossi & H.E. Freeman (1989). Evaluation: A systematic approach (4th ed.). Newbury Park, CA: 
Sage, p. 18. 

8. Conclusions of good or bad clearly are value judgments. Shadish and colleagues note that "Early 
evaluators mostly ignored the role of values in evaluation — whether in terms of justice, equality, liberty, 
human rights, or anything else. . . . such evaluators believed their activities could and should be value-free. 
But it proved to be impossible in the political world of social programming to evaluate without values 
becoming salient. Social programs are themselves not value-free." W.R. Shadish, Jr., T.D. Cook, & F.C. 
Feviton (1991). Foundations of program evaluation: Theories of practice. Newbury Park, CA: Sage, pp. 
46-47. 

9. Rossi & Freeman, Evaluation , p. 19. 

10. Scriven, Hard-won lessons in program evaluation , p. 44. 

11. R.E. Stake (1967). The countenance of educational evaluation. Teachers College Record , 68, 
523-40. Among program evaluators, Robert Stake is one of the early and long-term contributors. See 
Shadish, Cook, & Feviton, Foundations of program evaluation, for a comprehensive overview of his ideas 
and contribution, as well as those of other influential leaders such as Michael Scriven, Donald Campbell, 
Carol Weiss, Joseph Wholey, Fee Cronbach, and Peter Rossi. 

12. A relatively new form of evaluation practice is a process called "prospective evaluation," which has 
been developed by the Program Evaluation and Methodology Division (PEMD) of the U.S. General 
Accounting Office (GAO). The puipose of the process is to predict or forecast the impact of a proposed 
program or policy change (e.g., as an aid to legislators). The potential value of such forecasts is obvious; 
so are the problems associated with efforts to make accurate predictions. See General Accounting Office 
(1989). Prospective evaluation methods: The prospective evaluation synthesis. GAO/PEMD-89-10. 
Washington, DC: Author. 

13. Among academics, there is a running argument about the relative merits of quantitative and 
qualitative evaluations. In response to the many who argue primarily for quantitative evaluation, Guba and 
Fincoln have argued strongly for qualitative evaluation. See E.G. Guba & Y.S. Fincoln (1989). Fourth 
generation evaluation. Newbury Park, CA: Sage. 

Sechrest and Figueredo suggest that a compromise may be possible "in light of the realization that 
although rigorous theory testing is admittedly sterile and nonproductive without adequate theory 
development, creative theory construction is ultimately pointless without scientific verification." F. 
Sechrest & A. J. Figueredo (1993). Program evaluation. Annual Review of Psychology, 44, 645-74, p. 654. 

14. Posavac and Carey enumerate and discuss how political and psychological factors can undermine 
evaluation efforts, and suggest ways to plan for dealing with them. See E.J. Posavac & R.G. Carey (1989). 
Program evaluation: Methods and case studies (3rd ed.). Englewood Cliffs, NJ: Prentice-Hall. 
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1 5 . Thaip and Gallimore describe a fine example of program development based on a progressive series 
of formative and summative evaluations. Over a period of ten years, they made a succession of process and 
outcome evaluations using the quantitative data and qualitative information gathered on valuables affecting 
the outcomes to improve the program. That is, data gathered at each stage of program development were 
used as feedback for revising the intervention. See R.G. Thaip & R. Gallimore (1979). The ecology of 
program research and evaluation: A model for evaluation succession. In L. Sechrest, S.G. West, M.A. 
Phillips, R. Redner, & W. Yeaton (Eds.), Evaluation Studies Review Annual (Vol. 4, pp. 39-60). Beverly 
Hills, CA: Sage. 

16. Besides being difficult to carry out, evaluations of large-scale social and educational programs are 
costly, and the history of efforts to evaluate such programs is characterized by weak and often poorly 
conceived methodology as well as findings that are subject to varying inteipretations. At the same time, 
it is evident that such evaluations must be pursued, and we must learn to do them better. In this regard, 
each new national and state evaluation provides a unique opportunity to improve the process of evaluation. 

17. Charles Silberman cogently noted in his 1970 book, Crisis in the classroom (Vintage Books): 
"Elementary school students almost invariably regard mathematics as the most important subject in the 
curriculum — not because of its elegance, but because math has the most homework, because the homework 
is corrected the most promptly, and because tests are given more frequently than in any other subject. The 
youngsters regard spelling as the next most important subject, because of the frequency of spelling tests" 
(p. 147). 

We would add that, with increasing demands for accountability, teachers quickly learn what is evaluated 
and what is not, and slowly but surely greater attention is given to teaching what will be on the tests. Over 
time, what is on the tests becomes viewed as what is most important. Because only so much time is 
available to the teacher, other things not only are deemphasized, they also are dropped from the curriculum. 
If allowed to do so, accountability procedures have the power to reshape the entire curriculum. 

What’s wrong with that? Nothing — if what is evaluated reflects everything we want students to learn 
in school. Unfortunately, this is not the case. 

Current accountability pressures reflect values and biases that lead to evaluating a small range of basic 
skills and doing so in a narrow way. For students diagnosed with problems, this is seen in the fact that their 
school programs increasingly have been restricted to improving skills they lack. As a result, they are cut 
off from participating in learning activities that might increase their interest in overcoming their problems 
and that might open up opportunities and enrich their future lives. 

18. The issues related to the impact of a narrow focus on evaluation also arise in the context of 
discussions about evaluating intervener competence. That is, narrowly focused competency evaluations 
may constrict rather than expand intervener growth with respect to the broad range of knowledge, skills, 
and attitudes needed to properly plan, implement, and evaluate interventions. 

19. Cited in A. Smith. Supermoney. New York: Random House, p. 286. 
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2. Benchmarks for Monitoring and Reviewing Progress of Sustainability 
Activity 



Date Date Current Status 

started Completed 

I. Preparing the Argument 
for Sustaining Valued Functions 

Developing an understanding of the current status of 
the local big picture agenda 

Clarifying how specific functions have contributed to 
the big picture agenda (e.g., data on results) and 
where the functions fit in terms of current policy and 
program priorities 

Clarifying what valued functions will be lost if the 
school(s) and community do not determine ways to 
sustain them 



II. Mobilizing Interest, Consensus, and Date Date Current Status 

Support among Key Stakeholders started Completed 

Identifying champions for the functions and clarifying 
the mechanism(s) for bringing a broad base of 
supporters together to work on sustainability 

Clarifying cost-effective strategies for sustaining 
functions 

Planning and implementing a “social marketing” 
strategy specifically to garner a critical mass of 
stakeholder support 

Planning and implementing strategies to obtain the 
support of key policy makers 



III. Clarifying Feasibility Date Date Current Status 

started Completed 

Clarifying how the functions can be institutionalized 
into existing, modified, or new infrastructure of 
organizational and operational mechanisms 

Clarifying how necessary changes can be 
accomplished 

Formulating a longer-range strategic plan for 
maintaining momentum, progress, quality 
improvement, and creative renewal 
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IV. Proceeding with 

Specific Systemic Changes 


Date 

started 


Date 

Completed 


Current Status 


Assessing readiness to proceed with specific systemic 
changes 








Establishing an infrastructure and action plan for 
carrying out the changes 








Anticipating barriers and how to handle them 








Negotiating and renegotiating initial agreements 
(e.g., MOUs) 








Maintaining high levels of commitment to 
accomplishing desirable and necessary systemic 
changes 









An overarching benchmark involves the monitoring of the implementation of the evaluation action plan. 




Planners must understand 
the environment in which 
they work and acknowledge 
the chaos that is present.. 

W. Sybouts 
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